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MULTIRAD  Network  Performance  Evaluation  Plan 
During  F-15  Training  Utility  Evaluation  (TRUE) 

1.0  Introduction  &  Background 

Armstrong  Laboratory’s  Multiship  Research  and  Development  (MULTIRAD)  program  is  an 
effort  to  design,  develop  and  test  a  low-cost,  flexible  multiplayer  simulation  system  with  a  great 
degree  of  training  utility.  The  MULTIRAD  system  is  based  on  distributed,  low  cost, 
microprocessor  based  simulation  including  local  and  long  haul  networks. 

Armstrong  Laboratory’s  MULTIRAD  development  will  integrate  aircrew  training  simulators 
with  a  combat  environment  and  exercise  control  functions  on  an  asynchronous  network  based 
upon  the  SEMNET  version  6.6  protocol  with  extensions. 

The  F-15  Training  Requirements  Utility  Evaluation  (TRUE)  is  a  test  of  the  training  potential  of 
the  MULTIRAD  system.  As  part  of  the  F-15  TRUE,  the  MULTERAD  network  will  be 
monitored  and  its  performance  evaluated. 

For  TRUE  the  local  area  network  (LAN)  will  incorporate: 

•  Two  McDonnel  Douglas  Reconfigurable  Coclq)its  (MDRC)  configured  as  F-15C 
MSIP  cockpits, 

•  A  Ground  Control  Intercept  (GCI)  station, 

•  Two  Combat  Engagement  Trainers  (CET)  configured  as  F-16Cs, 

•  An  Automated  Threat  Environment  Simulator  (ATES), 

•  An  Exercise  Control  Station 

•  A  Data  Ex)gger 

For  the  long  haul  portion  of  TRUE,  the  network  wiU  be  connected  to  the  Institute  for  Defense 
Analysis  (IDA)  facility  in  Alexandria,  Virginia  where  an  F-15  Combat  Tactics  Trainer^  (CTT) 
and  GCI  station  will  be  added  to  the  network. 

Figure  1  is  a  block  Diagram  for  the  proposed  F-15  TRUE  Network. 

2.0  Objectives  of  the  F-15  TRUE  Network  Evaluation  Plan 

The  purpose  of  this  study  plan  is  to  document  the  F-15  TRUE  Network  Performance  Evaluation 
Plan.  This  plan  has  the  following  objectives: 

a) .  Perform  real-time  monitoring  of  network  traffic 

b) .  Identify  packet  type,  source  of  packet,  and  destination  of  packet 

c) .  Analyze  the  data  contained  within  packets 

d) .  Analyze  network  loading  characteristics 

e) .  Perform  comparison  of  multiple  packets 

f) .  Generate  packets  for  the  purpose  of  determining  protocol  compliance 
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3.1  General 

The  Network  Performance  Evaluation  will  occur  in  conjunction  with  TRUE  testing.  TRUE  is 
structured  around  simulated,  combat  oriented,  training  missions  for  F-15  pilots  and  Air  Weapons 
Controllers.  The  TRUE  will  consist  of  three  sets  of  tests  with  increasing  numbers  of  players 
and  simulation  complexity. 

The  test  scenarios  are: 

a) .  [1  vs  1]  -  one  F-15  vs  one- 

F-15  or 
GET  or 
AIT  or 

digitally  controlled  aircraft  or 
^  -  digitally  controlled  surface  threat 

This  test  wiU  focus  on  component  fidelity  rather  than  training  effectiveness 

b) .  [2  vs  "x"  (Local)]  -  Two  F-15s  vs  many  enemy  aircraft  including  CET’s,  ATT, 

digitally  controlled  aircraft  and  digitally  controlled  surface  threat 

c) .  [2  vs  "x"  (Long  haul)]  -  Two  F-15’s  vs  many  enemy  aircraft  with  players  located 

at  IDA. 


3.2  Target  Dates 

a) .  Complete  1  vs  1  preliminary  evaluations  -  TBD 

b) .  Conduct  1  vs  1  evaluation  by  F-15E  pilots  -  TBD  , 

c) .  Complete  2  vs  "x"  Gocal)  preliminary  evaluations  -  TBD 

d) .  Conduct  2  vs  "x"  (local)  scenarios  using  two  F-15E  pilots  -  TBD 

e) .  Conduct  2  vs  "x"  (long  haul)  scenarios  using  two  F-15C  pilots  plus  one  controller 

at  Brooks  AFB  and  2  F16  ATT  pilots  at  Luke  AFB  -  TBD 

f) .  Conduct  TRUE  for  2  vs  "x"  (local)  scenarios  using  16  F-15C  pilots  plus  8  AWCs 

-  TBD 

g) .  Conduct  TRUE  for  2  vs  "x"  (long  haul)  scenarios  using  16  F-15C  pilots  plus  8 

AWCs  -  TBD 

h) .  Draft  Report  on  TRUE  network  evaluation  -  TBD 

i) .  Final  report  on  TRUE  networl^  evaluation  -  15  DEC  92 
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4.0  TRUE  Performance  -  Data  Collection  and  Analysis  Procedure 
4. 1  Protocol  Compliance 

Prior  to  execution  of  the  full  TRUE  network  evaluation,  it  will  be  necessary  to  verify  the 
extensions  to  existing  Protocol  Data  Units  (PDU)  and  new  PDU-types  generated  under  the 

multirad  contract. 

The  extensions  to  existing  PDUs  are: 

Activate  Request 

•  Mission  Number 

•  Initial  Speed 

•  Initial  freeze  state 

•  Fuel  quantity 

•  Radio  channel 

Deactivate  Request 
^  •  Time  Stamp 

Vehicle  Appearance 

•  Fuel  quantity 

•  Throttle  position 

•  Linear  acceleration  vector 

•  Angular  velocity  vector 

Fire 

•  Time  Stamp 
Impact  , 

•  Time  Stamp 


New  PDUs  are: 

Radar 

Emitter 

Freeze 


The  protocol  extensions  and  new  PDUs  format  will  be  verified  by  stepping  a  simulator  through 
maneuvers  or  activating  controls  that  generate  the  specific  PDUs  and  then  capturing  and 
analyzing  the  network  traffic  using  the  data  logger. 


4.2  Network  Evaluation 


During  execution  of  the  actual  TRUE  exercises  i.e.  1  vs  1,  2  vs  "x"  "x"  Gong 

haul)  L  network  performance  wUl  be  recorded  and/or  analyzed  usmg  the  foUowmg  equipment. 


Data  logger/SUN  Workstation  -ipacket  by  packet  record  of  events 
Network  analyzer  -  Network  loading  and  performance  statistics 
NIU  network  data  and  queue  analysis 
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4.2.1  Data  Logger/SUN  Workstation 

The  network  PDU  traffic  for  each  exercise  will  be  recorded  by  a  data  logger  or  SUN 
Workstation  equipped  with  dual  tape  drives. 

The  data  logger  time-stamps  the  packets  as  it  receives  them  from  the  network  and  writes  them 
to  the  tape.  This  provides  a  permanent  record  of  events.  After  the  exercise  is  run,  the  data 
tapes  can  be  either  replayed  over  the  network  or  analyzed.  Play  back  is  visible  to  any  device 
on  the  network  (i.e.  Exercise  control  Station  and  simulators). 

The  data  logger,  when  set-up  in  the  continuous  record  mode  can  ping  pong  between  its  dual 
tape  drives  -  automaticaUy  switching  to  the  next  drive  when  the  first  one  is  fuU.  A  single  2400 
foot  tape  at  1600  bpi  will  record  between  one  and  ten  minutes  of  real  time  events  at  a  network 
data  rate  of  800,000  bps  (the  maximum  expected)  or  80,000  bps  respectively.  The  network  data 
rate  for  each  exercise  wiU  determine  whether  or  not  the  entire  exercise  is  recorded. 

Analysis  program^  for  the  data  tapes  are  being  written.  By  analyzing  the  tapes  off-line,  the 
following  things  can  be  determined: 

packet  latency 
#  of  corrupt  packets 

frequency  and  average  number  of  out-of-order  packets 

4.2. 1 . 1  Packet  Latency 

Maximum  and  average  packet  latency  is  determined  by  comparing  the  time  stamp  generated  at 
packet  creation  with  the  time  stamp  generated  by  the  data  logger  at  packet  receipt.  The  latency 
determined  here  wiU  represent  the  cumulative  delay  caused  by  waiting  in  the  NIU  output  queue 
for  access  to  the  ethemet  and  transmission  on  the  ethemet. 

Of  even  greater  interest  than  average  packet  latency  is  the  amount  of  jitter  or  variance  that  Ae 
network  experiences  from  packet  to  packet.  A  constant  delay  for  a  packets  arrival  may  not  be 
noticeable  whereas  a  variable  delay  may  cause  unacceptable  performance  degradation.  As  part 
of  the  post  analysis,  latency  jitter  wiU  be  measured  and  reviewed. 

4.2. 1.2  Packet  Corruption 

The  contents  of  a  packet  may  be  corrupted  during  transmission.  Packet  corruption  is  detenmned 
by  looking  at  the  data  fields  within  a  PDU  to  determine  if  all  values  are  withm  the  limits 

prescribed  by  the  Simnet  protocol. 

4.2. 1.3  Out-of-order  Packets 

Out-of-order  packets  can  occur  on  the  network  when  a  data  gram  sent  first  is  overtaken  by  one 
sent  second  because  it  encounters  a  larger  delay.  By  comparing  the  ongmating  time  stamps  on 
each  PDU,  statistics  for  out-of-order  packets  from  individual  sources  and  the  global  network  can 

be  determined. 
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4.2.2  Network  Analyzer 

The  network  analyzer  at  Armstrong  Labs  that  is  piggy  backed  on  a  Compaq  computer  can  be 
programmed  to  display  and  record  instantaneous  and  averaged  network  loading  information  such 
zs  bandwidth  utilization,  average  packet  size,  packet  type  frequency  of  occurrence  and  network 
anomalies  such  as  collisions. 

4.2.3  NIU  Data  Collection 

Each  NIU  will  need  to  run  background  programs  to  generate  performance  statistics  as  part  of 
their  normal  functioning.  These  statistics  will  indicate  how  much  idle  time  and  NIU  has  and 
how  big  the  input  and  output  PDU  queues  are  on  average  and  instantaneously.  Each  NIU  will 
also  need  to  monitor  the  frequency  of  time  late  data  both  globally  and  for  each  single  source. 
The  background  programs  needed  to  capture  this  data  and  generate  statistics  do  not  yet  exist  and 

must  be  written. 

4.3  Time  Synchronization 

In  order  to  accurately  time  stamp  each  PDU,  it  is  recommended  that  NIU’s  be  synchronized 
once  a  day  with  WWV.  WWV  is  a  radio  station  operated  by  the  National  Bureau  of  Stand^ds 
(NBS)  that  broadcasts  Coordinated  Universal  Timer  signals.  WWV  is  located  near  Fort  Collins, 
Colorado.  The  time  and  frequency  broadcasts  are  controlled  by  the  NBS  Frequency  Standard 
(a  cesium  atomic  standard).  The  frequencies  transmitted  by  WWV  are  held  stable  to  better  than 
2  part  in  1000  billion  at  all  times. 

Several  products  are  available  that  receive  the  WWV  broadcast  and  output  an  RS-232 
synchronization  signal.  The  RS-232  signal  would  then  be  tied  to  each  NIU  at  a  site.  For  *e 
long  haul  portion  of  the  testing,  IDA  and  WilHams  AFB  would  both  require  a  receiver  for 
synchronization.  The  synchronization  accuracy  achievable  by  using  WWV  broadcasts  would  be 
within  approximately  2030  milliseconds  for  a  long  haul  network  to  IDA.  An  alternate  method 
of  synchronizing  the  NIU’s  using  WWV  is  with  a  dial-up  system  that  automatically  m^ures 
the  round-trip  propagation  delay  on  the  telephone  and  compensates  for  it.  This  method  has  a 
clock  accuracy  of  10  milliseconds.  A  third  synchronization  method  would  be  to  broadcast  the 
freeze  command  over  the  net  from  a  single  NIU.  This  method  would  produce  a  synchronization 
accuracy  of  approximately  50  -  100  milliseconds  or  twice  the  total  net  propagation  delay,  which 

ever  is  greater. 

5.0  Content  of  Final  Report 

The  final  report  for  TRUE  Network  Evaluation  will  contain: 

a) .  Brief  description  of  the  simulation  system  networks 

b) .  Description  of  the  mission  scenarios 

c) .  Description  of  any  special  test  equipment 

d) .  Summary  of  network  perfqrmance  including: 

Peak  network  loading 
Average  network  loading 
Average  packet  size 
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Frequency  of  occurrence  of  packet  type 
#  of  network  message  collisions  detected 
Frequency  of  network  message  collisions 
Frequency  and  average  number  of  out-of-order  packets 

Analysis  of  network  loading  including: 

Individual  NIU  loading 
Maximum  NIU  queue  size 
Average  NIU  queue  size 
Maximum  message  latency 
Average  message  latency 
Analysis  of  packet  contents 
Packet  protocol  compliance 

Overall  assessment  of  network  performance  for  local  and  long  haul 
scenario’s 

Recommendations  for  future  network  improvements  and  developments 


